Limitations in Evaluating Machine Learning Models for Imbalanced Binary Outcome Classification in Spine Surgery: A Systematic Review

Clinical prediction models for spine surgery applications are on the rise, with an increasing reliance on machine learning (ML) and deep learning (DL). Many of the predicted outcomes are uncommon; therefore, to ensure the models’ effectiveness in clinical practice it is crucial to properly evaluate them. This systematic review aims to identify and evaluate current research-based ML and DL models applied for spine surgery, specifically those predicting binary outcomes with a focus on their evaluation metrics. Overall, 60 papers were included, and the findings were reported according to the PRISMA guidelines. A total of 13 papers focused on lengths of stay (LOS), 12 on readmissions, 12 on non-home discharge, 6 on mortality, and 5 on reoperations. The target outcomes exhibited data imbalances ranging from 0.44% to 42.4%. A total of 59 papers reported the model’s area under the receiver operating characteristic (AUROC), 28 mentioned accuracies, 33 provided sensitivity, 29 discussed specificity, 28 addressed positive predictive value (PPV), 24 included the negative predictive value (NPV), 25 indicated the Brier score with 10 providing a null model Brier, and 8 detailed the F1 score. Additionally, data visualization varied among the included papers. This review discusses the use of appropriate evaluation schemes in ML and identifies several common errors and potential bias sources in the literature. Embracing these recommendations as the field advances may facilitate the integration of reliable and effective ML models in clinical settings.


Introduction
In recent years, the integration of machine learning (ML) into spine surgery has shown promise in enabling personalized risk predictions [1,2].These advancements could improve patient outcomes, streamline surgical decision-making, reduce costs, and optimize medical management [3].ML, a subset of artificial intelligence (AI), utilizes computer algorithms to efficiently solve intricate tasks.A notable advantage lies in its adaptability, enabling models to continually learn and be redesigned by incorporating new data and modifying their underlying knowledge.
Machine learning has witnessed significant advancements, notably in the realm of deep learning (DL)-an advanced subset that involves neural networks with multiple layers, enabling more intricate data processing and abstraction.This structure contributes to its capability to automatically learn and extract features from complex datasets [4].The

Accuracy
Accuracy measures how well a model predicts the correct class.It is calculated as the ratio of correct predictions to the total number of predictions.However, when evaluating a binary classification model on an imbalanced dataset, accuracy can be misleading.This is because it only considers the total number of correct predictions without weighing the dataset's imbalance.
In scenarios with imbalanced datasets, a model consistently predicting the majority class can exhibit high accuracy but may struggle to accurately identify the minority class.When accuracy closely aligns with the class imbalance rate, it suggests the model might be predicting the majority class for all instances.In such cases, the accuracy is driven by the class imbalance, hindering the model's ability to distinguish between positive and negative classes.Therefore, it is crucial to employ multiple metrics for a comprehensive evaluation of the model's performance.

The Area under the ROC Curve (AUROC)
AUROC is calculated as the area under the curve of the true positive rate (TPR) versus the false positive rate (FPR).A no-skill classifier will have a score of 0.5, whereas a perfect classifier will have a score of 1.0.
While AUROC is useful for comparing the performance of different models, it can be misleading with class imbalance as the TPR and FPR are affected by the class distribution.
For instance, in a model predicting a specific disease on an imbalanced dataset, the TPR may be low as the model struggles to predict sick cases, while the FPR may be high because the model accurately predicts healthy cases.In such instances, the AUROC may yield falsely high-performance results.

Adequate Evaluation Metrics
In assessing a binary classification model on an imbalanced dataset, key metrics include the confusion matrix (CM), F1 score, Matthews correlation coefficient (MCC), and area under the precision-recall curve (AUPRC).

Confusion Matrix
The CM matrix delineates true positive, true negative, false positive, and false negative in model predictions [7].This matrix is particularly useful for imbalanced classes, offering insights into the model's performance on each class separately.It also facilitates the calculations of various metrics such as precision, recall, and F1 score.
As mentioned earlier, relying solely on accuracy is advised against in imbalanced cases, with the confusion matrix providing a strong rationale for that.Researchers can use it to visualize the model's performance, pinpoint common errors, and make the necessary adjustments to enhance overall performance.Table 1 displays the metrics provided by the CM.

True Positive (TP)
The number of predictions where the classifier correctly predicts the positive class as positive.
True Negative (TN) The number of predictions where the classifier correctly predicts the negative class as negative.

False Positive (FP)
The number of predictions where the classifier incorrectly predicts the negative class as positive.
False Negative (FN) The number of predictions where the classifier incorrectly predicts the positive class as negative.

Recall/Sensitivity
The proportion of true positive predictions to all actual positive cases TP/(TP + FN).

Specificity
The proportion of all negative samples that are correctly predicted as negative by the classifier TN/(TN + FP).
Precision/Positive predictive value (PPV) The proportion of true positive predictions to all positive predictions TP/(TP + FP).

Negative predictive value (NPV)
The proportion of true negative predictions to all negative predictions made by the model TN/(TN + FN).

F1 Score
Improving the model's performance often involves aiming for a balance between precision and recall.However, it is essential to acknowledge that there is a trade-off between these two metrics, where enhancement of one metric score can lead to a reduction in the other.The correct balance is highly reliant on the model's objective and is referred to as the F1 score.The F1 score is particularly useful when faced with imbalanced classes as it emphasizes the harmonic mean between precision and recall [8].

Matthews Correlation Coefficient (MCC)
The Matthews correlation coefficient (MCC) stands out as a robust metric, especially when dealing with imbalanced class data.MCC is a balanced metric that takes into account all four components of the CM.It is defined as (TP × TN − FP × FN)/sqrt((TP + FP) × (TP + FN) × (TN + FP) × (TN + FN)).The MCC tends to approach +1 in cases of perfect classification and −1 in instances of entirely incorrect classification (inverted classes).When facing class-imbalanced data, the MCC is considered a strong metric due to its effectiveness in capturing various aspects of classification performance.Notably, it remains close to 0 for completely random classifications.

Informedness (Youden's J Statistic)
Informedness, also known as Youden's J statistic, quantifies the difference between the true positive rate (Recall) and the false positive rate (FPR).It is computed as Recall + Specificity − 1, with values ranging from −1 to +1.A higher informedness value signifies a superior classifier [9].

Markedness
Markedness gauges the difference between the PPV and NPV.The calculation involves adding PPV and NPV, then subtracting 1, resulting in a range from −1 to +1.A higher markedness value suggests a better overall performance in predictive values [9].

The Area under the Precision-Recall Curve (AUPRC)
AUPRC is a valuable metric when working with imbalanced datasets as it considers precision and recall in its calculation [10].This is important when dealing with imbalanced datasets where the focus is on identifying positive cases and minimizing false positives.The AUPRC is derived by plotting precision and recall values at various thresholds and then computing the area under the resulting curve.
The resulting curve is formed by different points, and classifiers performing better under different thresholds will be ranked higher.On the plot, a no-skill classifier manifests as a horizontal line with precision proportional to the number of positive examples in the dataset.Conversely, a point in the top right corner signifies a perfect classifier.

Brier Score (BS)
The Brier Score (BS) serves as a metric for assessing the accuracy of a probabilistic classifier and is used to evaluate the performance of binary classification models [11].It is determined by calculating the mean squared difference between the predicted probabilities for the positive class and the true binary outcomes.The BS ranges from 0 to 1, with a score of 0 indicating a perfect classifier, while 1 suggests predicted probabilities completely discordant with actual outcomes.
It is important to note that while the BS possesses desirable properties, it does have limitations.For instance, it may favor tests with high specificity in situations where the clinical context requires high sensitivity, especially when the prevalence is low [12].
To address these limitations, a model's BS evaluation should consider the outcome prevalence in the patient sample, prompting the computation of the null BS.The null BS acts as a benchmark for evaluating a model's performance by always predicting the most prevalent outcome in the dataset.The model's BS is then compared to that of the null model, and ∆Brier is calculated by subtracting the null BS from that of the model under evaluation.The ∆Brier is a scalar value and indicates the extent to which the model outperforms the null model.The formula follows ∆Brier = BS of the model − BS of the null model.

Calibration Curves
A calibration plot is a graphical tool used to evaluate a probabilistic model.The curve illustrates the alignment between the model's predicted probabilities and the observed frequencies of the positive class in the test set.A perfect model would exhibit an intercept value of 0 and a slope value of 1.These plots are particularly valuable for evaluating models trained on imbalanced data, offering insights into the model's ability to predict the positive class.
Addressing imbalanced data involves using techniques such as undersampling and oversampling to achieve classification balance and alleviate classifier bias.However, determining the optimal sample size for training remains a significant challenge.An alternative strategy is to leverage learning curves, which provide insights into reducing error probability as the training set size increases.One example is a theoretical learning curve for the multi-class Bayes classifier, considering general multivariate parametric models of class-conditional probability density [13].This curve offers an estimate of the reduction in the excess probability of error without relying on specific model parameters.Learning curves contribute to an essential understanding of the model's behavior and its performance improvements with increased data.Table 1 outlines the metrics derived from the confusion matrix.

Decision Curve
A decision curve is a graphical tool used to evaluate a classifier's performance by examining the trade-off between sensitivity and 1-specificity across varying thresholds for classifying an instance as positive.The optimal threshold is the one that maximizes the net benefit.By convention, the model's benefit strategy at each threshold is compared to the treat-all and treat-none strategies.The decision curve analysis stands out from other statistical methods by its ability to evaluate the clinical value of a predictor.Figure 1A- probability of error without relying on specific model parameters.Learning curves contribute to an essential understanding of the model's behavior and its performance improvements with increased data.Table 1 outlines the metrics derived from the confusion matrix.

Decision curve
A decision curve is a graphical tool used to evaluate a classifier's performance by examining the trade-off between sensitivity and 1-specificity across varying thresholds for classifying an instance as positive.The optimal threshold is the one that maximizes the net benefit.By convention, the model's benefit strategy at each threshold is compared to the treat-all and treat-none strategies.The decision curve analysis stands out from other statistical methods by its ability to evaluate the clinical value of a predictor.Figure 1A-D depicts the AUROC, AUPRC, calibration, and decision curve figures.With that in mind, this systematic review of the literature aims to provide a comprehensive summary of the state of AI within the field of spine surgery.The focus will be on reporting metrics, data visualization, and common errors, including inappropriate handling of imbalanced datasets and incomplete reporting of model performance metrics.With that in mind, this systematic review of the literature aims to provide a comprehensive summary of the state of AI within the field of spine surgery.The focus will be on reporting metrics, data visualization, and common errors, including inappropriate handling of imbalanced datasets and incomplete reporting of model performance metrics.

Data Sources and Search Strategies
A comprehensive search of several databases was performed on 28 February 2023.Results were limited to the English language but had no date limitations.The databases included Ovid MEDLINE(R), Ovid Embase, Ovid Cochrane Central Register of Controlled Trials, Ovid Cochrane Database of Systematic Reviews, Web of Science Core Collection via Clarivate Analytics, and Scopus via Elsevier.The search strategies were designed and conducted by a medical librarian in collaboration with the study investigators (Table S1).Controlled vocabulary supplemented with keywords was used.The actual strategies listing all search terms used and how they are combined are available in the Supplemental Material.Ultimately, 3340 papers and 121 full-text articles were assessed, resulting in the inclusion of 60 studies (Figure 2) .This review was conducted in accordance with the PRISMA guidelines (Table S2).

Data Sources and Search Strategies
A comprehensive search of several databases was performed on 28 February 2023.Results were limited to the English language but had no date limitations.The databases included Ovid MEDLINE(R), Ovid Embase, Ovid Cochrane Central Register of Controlled Trials, Ovid Cochrane Database of Systematic Reviews, Web of Science Core Collection via Clarivate Analytics, and Scopus via Elsevier.The search strategies were designed and conducted by a medical librarian in collaboration with the study investigators (Table S1).Controlled vocabulary supplemented with keywords was used.The actual strategies listing all search terms used and how they are combined are available in the Supplemental Material.Ultimately, 3340 papers and 121 full-text articles were assessed, resulting in the inclusion of 60 studies (Figure 2) .This review was conducted in accordance with the PRISMA guidelines (Table S2).

Eligibility Criteria and Data Extraction
Inclusion criteria encompass studies focusing on ML-based prediction models pertaining to binary surgical outcomes following spine surgery.Both intraoperative and postoperative outcomes were eligible.Exclusion criteria comprised studies predicting nonbinary outcomes (e.g., 3+ categorical or numeric outcomes), those predicting non-spine surgical outcomes, studies with balanced outcomes, and those lacking predictive models.

Eligibility Criteria and Data Extraction
Inclusion criteria encompass studies focusing on ML-based prediction models pertaining to binary surgical outcomes following spine surgery.Both intraoperative and postoperative outcomes were eligible.Exclusion criteria comprised studies predicting nonbinary outcomes (e.g., 3+ categorical or numeric outcomes), those predicting non-spine surgical outcomes, studies with balanced outcomes, and those lacking predictive models.
The extracted data from all studies included the first author, paper title, year of publication, spinal pathology and surgery type, sample size, outcome variable (the primary result being measured), imbalance percentage, accuracy, AUROC (area under the receiver operating characteristic curve), sensitivity, specificity, PPV (positive predictive value), NPV (negative predictive value), Brier score (BS), other metrics, dataset, performance, journal, and error type (Table 2).

Data Synthesis and Risk of Bias Assessment
Our aim was to investigate the methodologies employed by the included studies, emphasizing the process rather than the outcomes or findings themselves.Accordingly, we refrained from engaging in narrative synthesis, data pooling, risk of bias assessment, or evidence certainty determination.Instead, our review specifically addressed methodologies related to models handling class imbalance.

Statistical Analysis
Given the considerable heterogeneity between studies, we did not perform a metaanalysis and opted for a qualitative and comprehensive analysis instead.Study characteristics are presented using frequencies and percentages for categorical variables.In cases where studies reported multiple results within a single outcome (e.g., different AUCs per type of complication), the top scores were taken.Metrics were computed for studies that provided a confusion matrix.

Characteristics of the Included Studies
The selected papers cover a variety of outcomes, some focusing on a single target while others address multiple targets.Table 2 outlines the metrics derived from the confusion matrix.Among the 60 papers, 12 focused on readmissions, 13 predicted lengths of stay (LOS), 12 addressed non-home discharge, 6 estimated mortality, and 5 anticipated reoperations.The models also forecasted a variety of medical and surgical outcomes, as detailed in Table 3.The target outcomes exhibited data imbalances ranging from 0.44% to 42.4%.Figure 3 illustrates the growing number of papers in the field over time.In the analysis of the 60 included papers, 59 reported the model's AUROC, 28 mentioned accuracies, 33 provided sensitivity, 29 discussed specificity, 28 addressed PPV, 24 considered NPV, 25 indicated BS (with 10 providing null model Brier), and 8 detailed the F1 score.Additionally, a variety of representations and visualizations were presented in these papers: 52 included an AUROC figure, 27 featured a calibration curve, 13 displayed a confusion matrix, 12 showcased decision curves, 3 incorporated PR curves, and only 1 In the analysis of the 60 included papers, 59 reported the model's AUROC, 28 mentioned accuracies, 33 provided sensitivity, 29 discussed specificity, 28 addressed PPV, 24 considered NPV, 25 indicated BS (with 10 providing null model Brier), and 8 detailed the F1 score.Additionally, a variety of representations and visualizations were presented in these papers: 52 included an AUROC figure, 27 featured a calibration curve, 13 displayed a confusion matrix, 12 showcased decision curves, 3 incorporated PR curves, and only 1 offered a precision-recall curve.Moreover, to train their models, 23 studies utilized NSQIP data, and 19 used single-center data, while the rest used multicenter data or other national datasets.In the following sections, we explore prevalent errors observed in the reviewed articles, highlighting key areas for improvement in the evaluation and reporting of machine learning models in spine surgery applications.[15].Similarly, Arora et al. developed a well-performing model that predicts patient discharge to rehabilitation, achieving high AUROC, sensitivity, and specificity with an adjusted threshold of 0.16 [32].Both studies also demonstrated well-calibrated models through calibration plots.
Shah et al. developed models predicting readmission or major complications, achieving satisfactory AUROC, AUPRC, and BS while outperforming the baseline AUPRC, indicating its effectiveness in predicting true positives well [17].Valliani et al. predicted non-home discharge with remarkable AUROCs, PPV, and NPV.The study also presented a well-calibrated model through a calibration plot, although the plot did not display true probability and predicted risks greater than 0.8 [18].Despite these models' solid performance on the metrics reported, studies in this category failed to report other metrics crucial for model evaluation.While some omitted the PPV and NPV, others failed to mention baseline AUPRC, sensitivity, specificity, and the null model BS.Without the inclusion of all the necessary evaluation metrics, the assessment lacks validity, even when reported metrics show high performance.

Error Type IIA: Metric Optimization at the Expense of Others
Li et al. developed artificial neural networks (ANN) and random forest (RF) models for predicting day-of-surgery patient discharge.The ANN model exhibited high sensitivity but low specificity, while the RF model showed the opposite [26].Kim et al. and Arvind et al. presented models predicting mortality, wound complications, venous thromboembolism, and cardiac complications [30,31,34].The Linear regression (LR) models exhibited high specificities at the expense of extremely low sensitivities.In contrast, ANN displayed high sensitivities and specificities but low PPVs.Goyal et al. developed models predicting non-home discharge and 30-day unplanned readmission [24].The models predicting nonhome discharge achieved high AUROCs, accuracies, sensitivity, specificity, and NPV but low PPV, leading to many false positives.This training method is advised only when the target is critically important and should not be missed, even if it means encountering many false positives.
Stopa et al. and Karhade et al. trained models to predict non-routine discharge, presenting high AUROC, BS, specificity, and NPV but low sensitivity and PPV [21,25].Although both models demonstrated well-calibrated performance via calibration plots, they struggled to detect positive cases correctly, facing low sensitivity scores and PPVs.Moreover, both papers presented a decision curve demonstrating that their models are better than the treat-all or the treat-non approach.

Error Type IIB: High Accuracy and AUROC but Poor Sensitivity
Cabrera et al. developed models that predict extended LOS, readmission, reoperation, infection, and transfusion.Although these models achieved high accuracies, their sensitivities were generally low, except for the model predicting transfusion [14].Gowd et al. predicted multiple surgical outcomes with high AUROCs and NPV but low PPV and extremely low sensitivity scores [19].Kalagara et al. trained models to predict unplanned readmission, reporting high accuracies but low sensitivities, while specificity, PPV, and NPV were not provided [22].Hopkins et al. developed a readmission prediction model with high accuracy, AUROC, specificity, PPV, and NPV but low sensitivity, indicating an inability to identify a significant proportion of true positive instances [23].

Other Errors
In addition to the previously mentioned errors, some papers provided poor calibration plots and omitted essential metrics.Kuris et al., Veeramani et al., and Zhang et al. presented models predicting readmission, unplanned re-intubation, and short LOS, respectively, with acceptable AUROCs, accuracies, and BSs [16,27,29].However, all three studies provided calibration plots indicating poor calibration, as the calibration curves were not in proximity to the near-perfect prediction diagonal.Moreover, the null model BS was not reported.Ogink et al. developed models predicting non-home discharge displaying adequate AUROCs and BSs [33].Nevertheless, the calibration plots in both studies revealed that the models were not well-calibrated for larger observed proportions and predicted probabilities, as the calibration curves drifted away from the near-perfect prediction diagonal.Furthermore, these five papers failed to report sensitivities, specificities, PPVs, and NPVs.

Discussion
ML's ability to predict future events by training on vast healthcare data has attracted substantial interest [73].Nevertheless, predicting rare events poses significant challenges attributed to the skewed data distribution.To address this issue, techniques for imbalanced class learning have been designed.This paper focuses on showcasing the application of ML in predicting uncommon patterns or events within the realm of spinal surgeries.These surgeries encompass various risks and require a thorough assessment of potential outcomes, such as readmission, reoperation, ELOS, and discharges to non-home settings [74,75].
We reviewed 60 papers addressing post-spinal surgery outcome predictions, examining specific elements of spinal surgeries such as pathologies, surgical procedures, and spinal levels.However, a limited number of these studies adequately evaluated their models using suitable metrics for imbalanced data binary classification tasks.This observation highlights the need for more rigorous model evaluation methods to ensure their clinical reliability and effectiveness in rare-event predictions.In a study by Haixiang et al., it was revealed that 38% of the 517 papers addressing imbalanced classification across various domains used accuracy as an evaluation metric despite the authors' awareness of dealing with an imbalanced problem [76].In some instances, the accuracy of a proposed method might be lower than the class imbalance ratio, implying that a dummy classifier solely predicting the majority class would yield better performance.
The importance of appropriate evaluation metrics for imbalanced classification problems in machine learning cannot be overstated.Our analysis revealed that many papers relied on inadequate evaluation metrics.Moreover, our review identified instances where models optimized one metric at the expense of others.These practices can lead to misinterpretation of model performance and hinder clinical applicability.Therefore, it is crucial to conduct a comprehensive evaluation of classifier performance, addressing all relevant metrics rather than focusing on only one or two.Additionally, striking a balance between the various performance metrics is essential to ensure that models can be effectively employed in clinical decision-making.By emphasizing the need for a holistic approach to classifier evaluation, our study encourages the development of more robust and reliable ML models for predicting rare outcomes in spinal surgery and other healthcare applications.
Training a binary classification model on an imbalanced dataset, where one class significantly outnumbers the other, poses challenges as the model may be biased towards the more prevalent class.Most strategies addressing this issue can be applied in the preprocessing stage prior to model training.These strategies include undersampling the majority class, oversampling the minority class, modifying weights, and optimizing thresholds.Another method for balancing classes is oversampling, which entails adding more minority class examples to the training dataset.For binary classification, strategies such as random oversampling, the synthetic minority over-sampling technique (SMOTE), and adaptive synthetic sampling (ADASYN) can be employed.Random oversampling adds random minority class samples to the training set until classes are equal, potentially leading to overfitting if the oversampled data does not represent the original minority class distribution.SMOTE, a more advanced technique, creates synthetic samples using the k-nearest neighbors algorithm to ensure new samples resemble original minority class samples [79].ADASYN is similar to SMOTE but generates synthetic samples more representative of the feature space region where the minority class is under-represented.While oversampling techniques appear more promising than undersampling ones, especially with small datasets, it is important to note that oversampling involves the addition of synthetic data that might not correspond to the real data.Given this constraint, advanced generative deep-learning algorithms were developed [80,81].One such advancement is generative adversarial network synthesis for oversampling (GANSO), which has demonstrated superior performance compared to the synthetic minority oversampling technique (SMOTE) [82].
In addition to the sampling methods discussed, threshold optimization can enhance classification model performance by adjusting the decision threshold for identifying positive category cases [83].This involves calculating the model's performance at various thresholds and selecting the one with the best performance.It is essential to conduct this optimization on a separate validation set to avoid overfitting.Once the optimal threshold is determined, it can be applied to a model's predictions on new data.
It is good practice to systematically test various suitable algorithms for the task at hand.Decision tree algorithms, such as random forest (RF), classification and regression tree (CART), and C4, perform well with imbalanced datasets.Additionally, classifiers' performance can be enhanced by assigning weights based on the inverse of class frequencies or using advanced techniques like cost-sensitive learning.In place of traditional classification models, anomaly detection models can also be used.Ensemble methods, such as bagging and boosting, are also effective in handling imbalanced data.Finally, it is crucial to evaluate using appropriate metrics for imbalanced classification tasks, such as MCC, CM, precision, recall, F1 score, and AUPRC.By employing a diverse set of metrics and considering the unique characteristics of each dataset, researchers can avoid being misled by metrics like accuracy and AUROC.

Conclusions
This systematic review summarizes the current literature on ML and DL in spine surgery outcome prediction.Evaluating these models is crucial for their successful integration into clinical practice, especially given the imbalanced nature of spine surgery predicted outcomes.The 60 papers reviewed focused on binary outcomes such as ELOS, readmissions, non-home discharge, mortality, and reoperations.The review highlights the prevalent use of the AUROC metric in 59 papers.Other metrics like sensitivity, specificity, PPV, NPV, Brier score, and F1 score were inconsistently reported.
Based on the findings of this review, our recommendations for future research in ML applications for spine surgery are threefold.First, we advocate for the comprehensive use and reporting of all appropriate evaluation metrics to ensure a holistic assessment of model performance.Second, developing strategies to optimize classifier performance on imbalanced data is crucial.Third, we stress the necessity of increasing awareness among researchers, reviewers, and editors about the pitfalls associated with inadequate model evaluation.To improve peer review quality, we suggest including at least one ML specialist in the review process of medical AI papers, as a high level of model design scrutiny is not a realistic demand from clinician reviewers.
The significance of proper evaluation schemes in applied ML cannot be overstated.Embracing these recommendations as the field advances will undoubtedly facilitate the integration of reliable and effective ML models in clinical settings.Ultimately, integrating such models in the clinical setting will contribute to improved patient outcomes, surgical decision-making, and medical management in spine surgery.

Figure 1 .
Figure 1.Illustrations of Various Performance Metrics for the Same Classifier: (A) Area Under the Receiver Operating Characteristic Curve, (B) Area Under the Precision-Recall Curve, (C) Calibration Curve, (D) Decision Curve.

Figure 1 .
Figure 1.Illustrations of Various Performance Metrics for the Same Classifier: (A) Area Under the Receiver Operating Characteristic Curve, (B) Area Under the Precision-Recall Curve, (C) Calibration Curve, (D) Decision Curve.

Figure 3 .
Figure 3. Annual Count of ML and DL Papers on Binary Outcome Prediction in Spine Surgery Included in the Review.

Figure 3 .
Figure 3. Annual Count of ML and DL Papers on Binary Outcome Prediction in Spine Surgery Included in the Review.
Undersampling involves reducing instances of the majority class in the training sample to equalize the classes.Various undersampling techniques, such as random undersampling, NearMiss, cluster-based undersampling, and Tomek links, can balance a dataset.Random undersampling selects a subset of majority class examples randomly, while NearMiss retains examples from the majority class closest to the minority class [77].Cluster-based undersampling sorts majority class examples into clusters and selects a representative subset from each cluster.Tomek links remove examples from the majority class closely related to minority class examples, increasing the space between classes and facilitating classification [78].

Table 1 .
Metrics Provided by the Confusion Matrix.
D depicts the AUROC, AUPRC, calibration, and decision curve figures.

Table 2 .
Performance Metrics, Datasets, and Outcome Variables in Reviewed ML Studies on Imbalanced Binary Classification in Spine Surgery.